Image capture apparatus, image processing system, and control method thereof

ABSTRACT

When distribution image data indicating a defocus distribution is generated from a plurality of sets of image data including pupil-divided image data that is captured image data corresponding to part of an exit pupil, reduced image data is used. Further, based on the distribution image data, pixel data that is to be subjected to the image processing is extracted from the pupil-divided image data corresponding to the distribution image data. The extracted pixel data, the defocus distribution image data, and captured image data corresponding to the entirety of the exit pupil are recorded or output to an external device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 14/699,702,filed Apr. 29, 2015 the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image capture apparatus, an imageprocessing system, and a control method thereof.

Description of the Related Art

Conventionally, there is known to be an image processing system in whicha defocus distribution of a captured image is detected and imageprocessing for locally changing a parameter corresponding to the defocusamount is applied to the captured image.

For example, Japanese Patent Laid-Open No. 2008-15754 discloses an imagecapture apparatus that includes an image sensor capable of generating apair of images from which a defocus amount can be calculated, andlocally adds a bokeh amount corresponding to the defocus amount to acaptured image.

In the image capture apparatus disclosed in Japanese Patent Laid-OpenNo. 2008-15754, two types (a pair) of image data generated from theoutputs of photoelectric conversion units of the same type are recordedwhen a captured image is recorded in response to a shutter release.Accordingly, in the case of using an image sensor with a configurationincluding two types of photoelectric conversion units per pixel, datasize to be recorded doubles as compared to the case of using a normalimage sensor having one photoelectric conversion unit per pixel.Therefore, the number of images that can be captured decreases comparedto the case of using the normal image sensor. Further, in the case whereimage processing is performed by an image processing apparatus separatefrom the image capture apparatus, the time for data communicationbetween the apparatuses increases. Further, since the data size doubles,the processing load required for data processing such as calculation ofthe defocus amount also increases.

The same problem occurs also in the case of using a pair of sets ofimage data captured by a stereo camera, or a pair of sets of image datacaptured at different focus distances. In recent years, with an increasein the number of pixels in image sensors, there are image sensors havingmore than 30,000,000 pixels, and therefore increases in data size arehaving a greater and greater influence.

SUMMARY OF THE INVENTION

The present invention aims to lighten the load of image processingcorresponding to a defocus distribution obtained from a plurality ofimages and to provide an image capture apparatus and an image processingsystem capable of improving at least one of the problems of theconventional techniques.

According to an aspect of the present invention, there is provided animage capture apparatus comprising: a reduction unit configured toreduce a plurality of sets of captured image data includingpupil-divided image data, which is captured image data corresponding topart of an exit pupil, and generate a plurality of sets of reduced imagedata; a generation unit configured to, on at least a pair of theplurality of sets of reduced image data that include reduced image dataof the pupil-divided image data, perform processing in which a pair ofsets of reduced image data are used to generate distribution image datathat indicates a spatial defocus distribution; an extraction unitconfigured to, based on the distribution image data, extract data ofpixels that are to be subjected to predetermined image processing fromthe pupil-divided image data corresponding to the distribution imagedata; and an output unit configured to output the extracted pixel data,the distribution image data, and captured image data corresponding tothe entirety of the exit pupil.

According to another aspect of the present invention, there is providedan image capture apparatus comprising: a generation unit configured to,on at least a pair of a plurality of sets of captured image dataincluding pupil-divided image data, which is captured image datacorresponding to a part of an exit pupil, perform processing in which apair of sets of captured image data are used to generate distributionimage data indicating a spatial defocus distribution; a reduction unitconfigured to, from the distribution image data, generate distributionimage data indicating a defocus distribution with a lower density thanthe distribution image data; an extraction unit configured to extractpixel data having a defocus amount as a target of a predetermined imageprocessing from pixel data constituting the distribution image datagenerated by the generation unit; and an output unit configured tooutput the extracted pixel data, the distribution image data generatedby the reduction unit, and captured image data corresponding to theentirety of the exit pupil.

According to still another aspect of the present invention, there isprovided an image capture apparatus comprising: a reduction unitconfigured to reduce a plurality of sets of captured image dataincluding pupil-divided image data, which is captured image datacorresponding to a part of an exit pupil, and generate a plurality ofsets of reduced image data; a generation unit configured to, on at leasta pair of the plurality of sets of reduced image data that includereduced image data of the pupil-divided image data, perform processingin which a pair of sets of reduced image data are used to generatedistribution image data that indicates a spatial defocus distribution;and an output unit configured to output the distribution image data andthe plurality of sets of captured image data.

According to yet another aspect of the present invention, there isprovided method for controlling an image capture apparatus, comprising:a reduction step of reducing a plurality of sets of captured image dataincluding pupil-divided image data, which is captured image datacorresponding to part of an exit pupil, and generating a plurality ofsets of reduced image data; a generation step of performing processingin which a pair of sets of reduced image data are used to generatedistribution image data indicating a spatial defocus distribution, on atleast a pair of the plurality of sets of reduced image data that includereduced image data of the pupil-divided image data; an extraction stepof extracting, based on the distribution image data, pixel data that isto be subjected to a predetermined image processing from thepupil-divided image data that corresponds to the distribution imagedata; and an output step of outputting the extracted pixel data, thedistribution image data, and captured image data corresponding to theentirety of the exit pupil.

According to still yet another aspect of the present invention, there isprovided a method for controlling an image capture apparatus,comprising: a generation step of performing processing in which a pairof sets of captured image data are used to generate distribution imagedata indicating a spatial defocus distribution, on at least a pair of aplurality of sets of captured image data including pupil-divided imagedata that is captured image data corresponding to a part of an exitpupil; a reduction step of generating, from the distribution image data,distribution image data indicating a defocus distribution with a lowerdensity than the distribution image data; an extraction step ofextracting pixel data having a defocus amount as a target of apredetermined image processing from pixel data constituting thedistribution image data generated in the generation step; and an outputstep of outputting the extracted pixel data, the distribution image datagenerated in the reduction step, and captured image data correspondingto the entirety of the exit pupil.

According to yet still another aspect of the present invention, there isprovided a method for controlling an image capture apparatus,comprising: a reduction step of reducing a plurality of sets of capturedimage data including pupil-divided image data that is captured imagedata corresponding to part of an exit pupil so as to generate aplurality of sets of reduced image data; a generation step of applyingprocessing in which a pair of sets of reduced image data are used togenerate distribution image data indicating a spatial defocusdistribution, on at least a pair of the plurality of sets of reducedimage data that include reduced image data of the pupil-divided imagedata; and an output step of outputting the distribution image data andthe plurality of sets of captured image data.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1C are block diagrams showing examples of functionalconfigurations of an image processing system, a digital camera, and apersonal computer according to an embodiment.

FIG. 2 is a diagram showing an example of a pixel array of an imagecapturing unit according to the embodiment.

FIG. 3 is a flowchart illustrating the operation of an image processingsystem according to a first embodiment.

FIG. 4 is a block diagram showing an example of a functionalconfiguration of an image processing unit 107 in FIG. 1B.

FIG. 5 is a block diagram showing an example of a functionalconfiguration of an image processing unit 206 in FIG. 1C.

FIG. 6 is a block diagram showing another example of the functionalconfiguration of the image processing unit 107 in FIG. 1B.

FIG. 7 is a block diagram showing an example of a functionalconfiguration of an image processing unit 107 according to a secondembodiment.

FIG. 8 is a block diagram showing an example of a functionalconfiguration of an image processing unit 206 according to the secondembodiment.

FIG. 9 is a diagram showing another example of the pixel array of theimage capturing unit according to the embodiment.

FIG. 10 is a block diagram showing another example of the functionalconfiguration of the image processing unit 107 in FIG. 1B.

FIG. 11 is a block diagram showing another example of the functionalconfiguration of the image processing unit 206 in FIG. 1C.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

First Embodiment

Hereinafter, an exemplary embodiment of the present invention will bedescribed in detail with reference to the drawings. The followingembodiment will describe an example in which the present invention isapplied to an image processing system 10, in which a digital camera 100serving as an example of an image capture apparatus and a personalcomputer (PC) 200 serving as an example of an image processing apparatusare communicably connected to each other, as shown in FIG. 1A. However,the processing performed by the PC 200 in the following description maybe executed by the digital camera 100. Further, the digital camera 100may be any electronic device having an image capturing function, and thePC 200 also may be any electronic device capable of performing theprocessing described below. In the following description, “adding bokeh”to an image means reducing the sharpness of the image or blurring theimage.

FIG. 1B is a block diagram showing an example of a functionalconfiguration of the digital camera 100 according to this embodiment. Acontrol unit 101, for example, is a CPU, and controls the operation ofeach block included in the digital camera 100 by reading out a programfrom a ROM 102, deploying it to a RAM 103, and executing it. The ROM 102is a rewritable non-volatile memory, and stores parameters and the likethat are necessary for the control of the blocks, in addition to theprogram executed by the control unit 101. The RAM 103 is a rewritablevolatile memory, and is used as a temporary storage region for dataoutput by the blocks included in the digital camera 100.

An optical system 104 forms a subject image in an image capturing unit105. For example, the image capturing unit 105 is an image sensor, suchas a CCD or CMOS sensor, which photoelectrically converts the subjectimage formed by the optical system 104 and outputs the thus-obtainedanalog image signals to an A/D converter 106. The A/D converter 106applies A/D conversion processing to the input analog image signals andoutputs the obtained digital image data to the RAM 103.

An image processing unit 107 applies various types of image processing,such as white balance adjustment, demosaicing, reduction/enlargement,filtering, encoding, and decoding, to the image data stored in the RAM103.

A recording medium 108, for example, may be a detachable memory card,and is used for recording the image data processed by the imageprocessing unit 107, the image data output from the A/D converter 106,and the like, which are stored in the RAM 103, as an image data file ofa predetermined format. A communication unit 109 transmits the imagedata file or the like that is recorded in the recording medium 108 to anexternal device by wire or wirelessly.

A display unit 110 displays image data obtained by shooting or imagedata read out from the recording medium 108, or displays various menuscreens. It also functions as an electronic viewfinder by displaying alive view image.

An operation unit 111 is an input device group for allowing a user toinput various instructions, settings, and the like to the digital camera100, and includes keys or buttons that are included in a general digitalcamera, such as a shutter button, a menu button, a direction key, and adetermination key. Further, in the case where the display unit 110 is atouch display, the display unit 110 serves also as the operation unit111. The operation unit 111 may have a configuration that does notrequire physical operation, such as a combination of a microphone and avoice command recognition unit.

FIG. 1C is a block diagram showing an example of a functionalconfiguration of the PC 200 according to this embodiment. A control unit201, for example, is a CPU, and controls the operation of each blockincluded in the PC 200 by reading out a program from a ROM 202,deploying it to a RAM 203, and executing it. The ROM 202 is a rewritablenon-volatile memory, and stores parameters and the like that arenecessary for the control of the blocks in addition to the programexecuted by the control unit 201. The RAM 203 is a rewritable volatilememory, and is used as a temporary storage region for data output by theblocks included in the PC 200.

A communication unit 204 communicates with an external device such asthe digital camera 100 via wired or wireless communication. A recordingdevice 205, for example, is a hard disk, and stores image data receivedby the communication unit 204 from the digital camera 100 or the like.

An image processing unit 206, for example, applies image processingcorresponding to a later-described defocus amount to the image datadeployed from the recording device 205 to the RAM 203.

A display unit 207 is used for displaying various types of data or a GUIprovided by an OS or an application operating in the PC 200. The displayunit 207 may be included in the PC 200, or may be connected thereto asan external device.

An operation unit 208 is an input device group for allowing the user toinput various instructions, settings, and the like to the PC 200, andusually includes a keyboard, a mouse, a track pad, and the like.Further, in the case where the display unit 207 is a touch display, thedisplay unit 207 serves also as the operation unit 208. The operationunit 208 may have a configuration that does not require physicaloperation, such as a combination of a microphone and a voice commandrecognition unit.

FIG. 2 is a schematic view showing an example of a pixel array of theimage capturing unit 105. The image capturing unit 105 of thisembodiment has a plurality of pixels that are two-dimensionally arrayed,and each pixel 502 has a microlens 501 and a pair of photoelectricconversion units 503 and 504. Here, image data that is obtained from thephotoelectric conversion unit 503 group and corresponds to the rightfocus detection pupil is expressed as R (x, y), and image data that isobtained from the photoelectric conversion unit 504 group andcorresponds to the left focus detection pupil is expressed as L (x, y)(where x and y are integers of 0 or more). The image capturing unit 105of this embodiment outputs R (x, y) and basic image data g (x, y) in thefollowing formula as a pair of captured image data sets.

g(x,y)=R(x,y)+L(x,y)  (1)

That is, the image capturing unit 105 outputs image data obtained by thephotoelectric conversion unit 503 group, and image data obtained byadding the image data obtained by the photoelectric conversion unit 503group and the image data obtained by the photoelectric conversion unit504 group. The basic image data g (x, y) is image data that correspondsto the entirety of the exit pupil.

The pair of sets of image data output by the image capturing unit 105may be another combination. For example, a combination of L (x, y) and g(x, y) or a combination of R (x, y) and L (x, y) may be employed.

Hereinafter, the operation of an image processing system of thisembodiment will be described with reference to FIG. 3 to FIG. 5.

FIG. 4 is a diagram schematically showing functions of the imageprocessing unit 107. In FIG. 4, functional blocks corresponding tovarious types of image processing that can be applied by the imageprocessing unit 107 are depicted for convenience of description.However, in FIG. 4, the steps of image processing for applying a bokehamount corresponding to a defocus amount are shown schematically usingthe functional blocks and thus FIG. 4 does not necessarily match theactual configuration. For example, in the case where the imageprocessing unit 107 is a programmable processor such as a GPU or a DSP,the functional blocks may be implemented by software, and further, theinput-output lines between the functional blocks may be absent.

An image reducing unit 301 reduces the size of basic image data g (x, y)302 and image data R (x, y) 303 that corresponds to the right focusdetection pupil, which are stored in the RAM 103, to 1/N horizontallyand 1/N (N>1) vertically. The reducing method is not specificallylimited, and generally-known methods such as a bicubic method, forexample, can be used therefor. In this embodiment, it is assumed thatimage data before the reduction has 6400 pixels horizontally×4800 pixelsvertically, and N is 4. Accordingly, the image reducing unit 301generates reduced image data Rs (x, y) 305 and gs (x, y) 306 with 1600pixels horizontally×1200 pixels vertically from the R (x, y) 303 and theg (x, y) 302, and outputs them.

Based on the Rs (x, y) 305 and the gs (x, y) 306 that are output by theimage reducing unit 301, a Def distribution calculating unit 304 outputsimage data Defs (x, y) 309, which indicates a spatial defocusdistribution in the image capture range. A defocus amount is an amountof focus shift from the distance at which the optical system 104 is infocus, and thus is also distance information. In this embodiment, sincethe defocus amount is calculated using the reduced image data, theprocessing load of the Def distribution calculating unit 304 issignificantly lightened as compared to the case where the defocus amountis calculated with respect to the original image data. Accordingly, thedefocus amount can be calculated at a higher speed, the performancerequired of the Def distribution calculating unit 304 can be suppressed,and the circuit scale and power consumption can be reduced.

The Def distribution calculating unit 304 first calculates Ls (x, y) bysubtracting the Rs (x, y) 305 from the gs (x, y) 306.

Then, the Def distribution calculating unit 304 determines the spatialdefocus distribution by dividing the reduced image data Rs (x, y) and Ls(x, y), which correspond to the left and right focus detection pupilsrespectively, into a plurality of small blocks and calculating a defocusamount for each corresponding small block.

Data of m pixels is extracted from the Rs (x, y) and Ls (x, y) in thedetection direction of the defocus amount (for example, in the xdirection), and the respective data series are expressed as (E(1) toE(m)) and (F(1) to F(m)) in a generalized manner. The Def distributioncalculating unit 304 calculates a correlation amount C(k), when the dataseries (F(1) to F(m)) are displaced by a displacement amount k relativeto the data series (E(1) to E(m)), using the following Formula (2).

C(k)=Σ|E(n)−F(n+k)|  (2)

In Formula (2), Z is calculated with respect to n, and n and (n+k) arelimited to the range of 1 to m. The displacement amount is an integer,and is a relative shift amount obtained using the extraction pitch ofpixel data constituting the data series as a unit. Since this embodimentuses the image data reduced to 1/N in the horizontal and verticaldirections, the reduced image data needs to be multiplied by N in orderto be expressed in terms of the pitch of the image data before thereduction.

Letting the displacement amount k at which the discrete correlationamount C(k) is the minimum be kj, the Def distribution calculating unit304 can determine the displacement amount x at which a continuouscorrelation amount C(x) takes the minimum value, using the followingformulas.

x=(kj+D/SLOP)·N  (3)

D={C(kj−1)−C(kj+1)}/2  (4)

SLOP=MAX{C(kj+1)−C(kj),C(kj−1)−C(kj)}  (5)

Based on the displacement amount x determined by Formula (3), the Defdistribution calculating unit 304 can determine a defocus amount Defwith respect to a planned image forming surface of a subject imagesurface using Formula (6). Multiplication by N at the end of Formula (3)is performed in order to restore the extraction pitch of pixel data inthe reduced image data to the pitch in the image data before thereduction, and N is 1 when processing for detecting the defocus amountis performed without using the reduced image data.

Def=KX·PY·x  (6)

PY in Formula (6) is the extraction pitch of pixel data, to which thepitch in the image before the reduction is applied, regardless ofwhether or not the image is reduced. KX is a conversion factordetermined depending on the size of the divergence angle of thebarycenters or centers of the pair of focus detection pupils.

In the calculation of Formula (2) to Formula (6), the defocus amount iscalculated by allocating corresponding small blocks to the coordinates(x, y) of an input image, and therefore the defocus amount is calculatedfor each pixel (coordinate) of the input image. Such data indicating thedefocus distribution is called defocus distribution image data in thisdescription since it has the same data configuration as the input image(which is herein 1600 pieces of data horizontally x 1200 pieces of datavertically) and the defocus amount is also distance information.

Further, the defocus distribution image data output by the Defdistribution calculating unit 304 is generated from the reduced imagedata, and therefore is called reduced defocus distribution image dataDefs (x, y) 309. As described above, small blocks are arranged for eachpixel in the calculation of the defocus amount, and therefore smallblocks arranged for the pixel with coordinates (x, y) and small blocksarranged for the pixel with coordinates (x+1, y) partially overlap eachother, for example. For example, the small blocks can be arranged sothat corners of the small blocks serve as target pixels, but there is nolimitation to this.

A bokeh adding unit 308 adds bokeh corresponding to the defocus amountDef indicated by the reduced defocus distribution image data Defs (x, y)309 to the reduced basic image data gs (x, y) 306, and outputsbokeh-added reduced basic image data Vs (x, y) 311. Specifically, thebokeh adding unit 308 applies a convolution operation on reduction pointspread function PSF (x, y) by bokeh to the gs (x, y) 306, as shown inFormula (7), so as to generate bokeh-added reduced basic image data Vs(x, y) 311.

Vs(x,y)=gs(x,y)⋄PSF(x,y)  (7)

In Formula (7), ⋄ denotes a two-dimensional convolution operation. Forexample, the bokeh-added reduced basic image data Vs (x, y) 311 isdisplayed on the display unit 110 and used by the user for simplychecking the quality of the image resulting from image processing withthe digital camera 100 in FIG. 1A and FIG. 1B.

The reduction point spread function PSF (x, y) applied by the bokehadding unit 308, for example, may be as follows.

PSF(x,y)=1/(π·r̂2) if x̂2+ŷ2≤r̂2

PSF(x,y)=0 if x̂2+ŷ2>r̂2  (8)

In Formula (8), r denotes the radius of point image bokeh, where PSF (x,y)=δ (x, y) is true when r=0.

r=0 if |Def|≤Def0

r=Zf·(|Def|−Def0)/N if Def0<|Def|≤Def1

r=Zf·(Def1−Def0)/N if Def1<|Def|  (9)

Def1=M·Def0  (10)

In Formulas (9) and (10), Zf is a bokeh intensity parameter which ismanually input by operating the operation unit 111 or is stored inadvance in the ROM 102, for example. Def0 is a parameter indicating thedepth of field, and no bokeh is added to subject images within the depthof field. M is a parameter for determining the Def range within whichthe size of bokeh is changed, and any parameter greater than or equal to1 can designated therefor, but M is 2 in this embodiment. When thedefocus amount increases to some extent, it becomes difficult tovisually recognize a change of the radius of point image bokeh in thebokeh-added basic image, and therefore the radius of point image bokehwith a Def exceeding Def1 is not changed. Division by N in Formula (9)is performed because the pixel extraction pitch of the data series is Ntimes the value in the image before the reduction, and N is 1 when thereduced image is not used.

An image enlarging unit 313 enlarges the reduced defocus distributionimage data Defs (x, y) 309 so that it has the same number of pixels asthe R (x, y) 303, which is an extraction target of a region extractingunit 312, and outputs it as defocus distribution image data Defe (x, y)310. Here, the image enlarging unit 313 enlarges it N times horizontallyand N times vertically, conversely to the image reducing unit 301. Theenlarging method is not particularly limited, and generally-knownmethods such as a bicubic method, for example, can be used therefor.Increasing the number of pixels in the defocus distribution image isequivalent to increasing the density of the defocus distribution.

The region extracting unit 312 extracts a specific region from the imagedata R (x, y) 303 corresponding to the right focus detection pupil basedon the defocus distribution, or more specifically, the defocusdistribution image data Defe (x, y) 310, and outputs extracted imagedata Rp (x, y) 316. The specific region is a region that is to be atarget of image processing corresponding to the defocus amount.

For each pixel of the R (x, y), the region extracting unit 312determines whether or not to output (extract) the target pixel of the R(x, y) by referring to the defocus amount indicated by the correspondingpixel of the defocus distribution image data Defe (x, y).

Specifically, letting the defocus amount indicated by the Def (x, y) beDefm, the region extracting unit 312 performs determination such that:pixels satisfying |Defm|≤Def0 are not output; pixels satisfyingDef0<|Defm|≤Def1 are output; and pixels satisfying Def1<|Defm| are notoutput, and outputs the extraction data Rp (x, y).

In this embodiment, the reduced defocus distribution image data Defs (x,y) generated by the Def distribution calculating unit 304 is enlarged bythe image enlarging unit 313, but Defs (x, y) that is to be input intothe image enlarging unit 313 may be generated separately.

When a shooting instruction such as a full press of the shutter buttonis input through the operation unit 111, the digital camera 100 of thisembodiment executes processing at the time of recording shown in FIG. 3.The control unit 101 performs shooting processing in accordance withexposure conditions determined in a shooting preparation state, andobtains a pair of captured image data sets from the image capturing unit105. Here, as described above, the basic image data g (x, y) and one (R(x, y)) of the pupil-divided image data sets R (x, y) and L (x, y) areobtained as captured image data and are stored in the RAM 103 (S610).

Then, the aforementioned processing is applied using the imageprocessing unit 107. That is, a defocus distribution image Defs (x, y)is generated by the Def distribution calculating unit 304 from thereduced image data sets gs (x, y) and Rs (x, y) generated by the imagereducing unit 301 (S620). Then, based on the Defe (x, y) generated bythe image enlarging unit 313 enlarging the defocus distribution imageDefs (x, y), pixel data that is to be a target of image processing isextracted from the R (x, y), and thereby the extraction data Rp (x, y)is generated (S630).

Subsequently, the control unit 101 transmits the image data to the PC200 through the communication unit 109 (S640). In this embodiment,processing of step S650 and onward is performed by the PC 200, andtherefore will be described later. It should be noted that the followingimage data may be transmitted to the PC 200 through the communicationunit 109 after being recorded in the recording medium 108, or thefollowing image data may be read out by the PC 200 from the recordingmedium 108 detached from the digital camera 100.

-   -   Basic image data g (x, y) (6400×4800 pixels)    -   Reduced defocus distribution image data Defs (x, y) (1600×1200        pixels)    -   Extraction data Rp (x, y)

Although the amount of Rp (x, y) depends on the defocus distribution,pixels in a region within the depth of field and pixels in a region witha comparatively large defocus amount are excluded. Therefore, the numberof pixels is normally significantly lower than 6400×4500(=6400×4800−1600×1200) pixels.

Further, in FIG. 4, the basic image data g (x, y) is depicted as beingoutput from the image processing unit 107, but it may be read out fromthe RAM 103 by the control unit 101 at the time of recording ortransmission. The aforementioned three sets of image data may berecorded or transmitted collectively as one data file, or may berecorded or transmitted as separate files. However, in the latter case,it is necessary to make them recognizable as being associated datafiles, for example, by giving them the same file name or placing them inthe same folder. Since the file structure in recording or transmission,the transmission control corresponding to the transmission protocol, andthe like are not directly relevant to the present invention and knownmethods can be used therefor, detailed descriptions thereof will beomitted.

Accordingly, the amount of data to be recorded or transmitted can be cutin most cases, such as a case of transmitting:

-   -   the image data R (x, y) (6400×4800 pixels) corresponding to the        right focus detection pupil; and    -   the image data L (x, y) (6400×4800 pixels) corresponding to the        left focus detection pupil. Therefore, the number of images that        can be captured by the digital camera 100 can be increased, and        the time for communicating with an external device can be        further shortened, according to which it is possible not only to        improve the usability, but also to reduce the power consumption        of the digital camera 100.

FIG. 5 is a block diagram schematically showing an example of thefunctional configuration of the image processing unit 206 included inthe PC 200 of this embodiment. Hereinafter, the operation of the imageprocessing unit 206 will be described with reference to FIG. 3. Theoperation of the image processing unit 206 is realized in accordancewith the control of the control unit 201.

An image enlarging unit 402 enlarges reduced defocus distribution imagedata Defs (x, y) 403 so as to generate defocus distribution image dataDefe (x, y) in the same manner as the image enlarging unit 313 includedin the image processing unit 107 of the digital camera 100 (S650).Specifically, the image enlarging unit 402 generates the Defe (x, y) byenlarging the reduced defocus distribution image data Defs (x, y) 403 Ntimes horizontally and N times vertically so that the resolution (thenumber of pixels) thereof matches that of basic image data g (x, y) 405.The image enlarging unit 402 outputs defocus distribution image dataDefe (x, y) obtained by the enlargement to a Def distributioncalculating unit 401 and a bokeh adding unit 407.

For each pixel of the defocus distribution image data Defe (x, y), theDef distribution calculating unit 401 determines whether or not torecalculate (update) the defocus amount indicated by the pixel.

Specifically, letting the defocus amount indicated by pixel dataconstituting the defocus distribution image data be Defm, the Defdistribution calculating unit 401 performs determination such that:

when |Defm|≤Def0 is satisfied, Defm is not updated (Defm is output);when Def0<|Defm|≤Def1 is satisfied, Defm is updated; andwhen Def1<|Defm| is satisfied, Defm is not updated (Defm is output). TheDef0 and Def1 used herein are the same as those used in Formulas (9) and(10).

In this way, the defocus amount is not recalculated for pixels withinthe depth of field (|Defm|≤Def0) and pixels whose Def exceeds Def1, andthe defocus amount calculated in the digital camera 100 is used as-is.The range of the defocus amount to be recalculated (Def0<|Defm|≤Def1) isa range that is the target of the processing for adding bokeh (changingthe radius of point image bokeh) in the bokeh adding unit 407 and is arange that is the extraction target in the region extracting unit 312.

In this way, the defocus amount is recalculated only for the pixelshaving a defocus amount such that the radius of the point image bokeh ischanged by the bokeh adding unit 407, and therefore the defocusdistribution can be calculated at high speed. Further, since the defocusdistribution image data Defs (x, y) received from the digital camera 100is generated from the reduced image, the Defe (x, y) resulting from theenlargement performed by the image enlarging unit 402 also has a roughdefocus resolution (resolution of the displacement amount). Accordingly,a high defocus resolution is obtained by recalculating the defocusamount based on image data that has not been reduced for the targetwhose bokeh amount is adjusted corresponding to the defocus amount.

Hereinafter, processing for recalculating (updating) the defocus amountwill be described.

The Def distribution calculating unit 401 determines left focusdetection pupil image data Lp (x, y) corresponding to the Rp (x, y) fromthe basic image data g (x, y) and the extraction data Rp (x, y) receivedfrom the digital camera 100. Specifically, the Def distributioncalculating unit 401 obtains the Lp (x, y) by subtracting the Rp (x, y)from the g (x, y). As described above, the Rp (x, y) is obtained byextracting pixel data having a defocus amount that falls underDef0<|Defm|≤Def1 from the R (x, y).

The Def distribution calculating unit 401 calculates the correlationamount C(k) for each small block by performing the same processing asthat performed by the Def distribution calculating unit 304 on the Rs(x, y) and the Ls (x, y), on the pair of image data sets Rp (x, y) andLp (x, y).

At this time, the Def distribution calculating unit 401 refers to thedefocus amount Defm that is indicated by the defocus distribution imagedata Defe (x, y) supplied from the image enlarging unit 402, and adjustsa maximum value kmax of the displacement amount k for each pixel. FromFormula (6), kmax is obtained as follows.

kmax=Defm/(KX·PY)  (11)

PY is the extraction pitch of pixel data constituting the data seriesfor calculating the correlation amount, and KX is a conversion factordetermined by the divergence angle of the barycenters or centers of thepair of focus detection pupils.

Further, in consideration of the rough resolution of the Defm of the Def(x, y) supplied from the image enlarging unit 402, a fixed value k0 maybe added to Formula (11) as a margin. In this case, kmax is obtained asfollows.

kmax=Defm/(KX·PY)+k0  (12)

In either case, the range of the displacement amount k is adjusted foreach pixel so that kmax decreases as the defocus amount of the pixeldecreases, and therefore the time for calculating the correlation amountC(k) can be shortened for pixels having a small defocus amount.

Then, based on the correlation amount C(k) obtained with respect to thediscrete displacement amount k, the Def distribution calculating unit401 uses Formulas (3) to (5) to obtain a shift amount x that provides aminimum value C(x) with respect to a continuous correlation amount.Further, based on the shift amount x, the Def distribution calculatingunit 401 determines a defocus amount Def of the planned image formingsurface of the subject image surface in accordance with Formula (6).

Thus, the Def distribution calculating unit 401 recalculates the defocusamount of pixels that correspond to the extraction data Rp (x, y)(pixels to which bokeh is added by the bokeh adding unit 407), andgenerates updated defocus distribution image data Def′ (x, y) 406(S660).

The bokeh adding unit 407 has the same configuration as the bokeh addingunit 308 included in the image processing unit 107 of the digital camera100. For each pixel of the basic image data g (x, y) received from thedigital camera 100, the bokeh adding unit 407 refers to a defocus amountDef indicated by the corresponding updated defocus distribution imagedata Def′ (x, y) 406. Then, as shown in Formula (13), the bokeh addingunit 407 generates bokeh-added basic image data V (x, y) by applying aconvolution operation on reduction point spread function PSF (x, y)corresponding to the Def to the pixel value of the g (x, y) (S670).

V(x,y)=g(x,y)⋄PSF(x,y)  (13)

In Formula (13), ⋄ denotes a two-dimensional convolution operation.

The bokeh-added basic image data V (x, y) is recorded in the recordingdevice 205 in FIGS. 1C and 1 s used by the user for printing as a finalimage for an ornamental purpose or for uploading to an SNS so as toshare it with other users.

The reduction point spread function PSF (x, y) described in Formulas (8)to (10) can be used herein.

Although the bokeh adding unit 308 in FIG. 4 also performs the sameprocessing as that performed by the bokeh adding unit 407 in FIG. 5, thebokeh adding unit 308 generates the bokeh-added reduced basic image dataVs (x, y), and therefore the processing performance required for thebokeh adding unit 308 is lower than that required for the bokeh addingunit 407. Accordingly, the bokeh adding unit 308 can be configured tohave a smaller circuit scale and less power consumption than the bokehadding unit 407.

In this way, according to this embodiment, when a defocus amount iscalculated from a pair of captured image data sets obtained using animage sensor having a plurality of photoelectric conversion units foreach pixel, the use of image data having a number of pixels that is cut(reduced) can increase the calculation speed of the defocus amount.Further, also in the case where image processing corresponding to thedefocus amount is applied, the use of reduced image data for display canincrease the processing speed.

Further, the image data to be recorded is a combination of basic imagedata having one set of data per pixel, defocus image data indicating adefocus amount obtained from reduced image data, and data indicating aregion to which image processing corresponding to the defocus amount isto be applied. Therefore, the amount of the recording data can bereduced as compared to the case where data is recorded for eachphotoelectric conversion unit (the case where a plurality of data setsare recorded per pixel). Accordingly, the time required for recordingthe data to be recorded in a recording medium or transmitting it to theoutside can be shortened, and the number of images that can be capturedby the image capture apparatus can be increased.

Further, the processing load of the image capture apparatus can belightened by using an image processing apparatus separate from the imagecapture apparatus for processing the image data to be recorded. Further,the defocus distribution data and the data indicating a region to whichimage processing corresponding to the defocus amount is to be appliedare provided to the image processing apparatus, whereby it is possibleto lighten the processing load required for recalculation of the defocusamount in the image processing apparatus.

In this embodiment, the basic image data, the reduced defocusdistribution image data, and the extraction data are recorded ortransmitted to an external device by the digital camera 100. However,other data may be used as long as it is a combination of the basic imagedata, the defocus distribution data based on the reduced image data, andthe data indicating a region to which image processing corresponding tothe defocus amount is to be applied.

FIG. 6 is a block diagram showing another configuration example of theimage processing unit 107. Here, the Def distribution calculating unit304 generates defocus distribution image data Def (x, y) 317 from the g(x, y) 302 and the R (x, y) 305. Then, defocus distribution image dataDefp (x, y) data extracted from the Def (x, y) by the region extractingunit 312 is output as the data indicating a region to which imageprocessing corresponding to the defocus amount is to be applied. In thiscase, although the load for calculating the defocus amount cannot belightened since reduced image data is not used, the need for the imageenlarging unit 313 is eliminated. Further, the image reducing unit 301reduces the Def (x, y) generated by the Def distribution calculatingunit 304, so as to generate the Defs (x, y) 309. Reducing the number ofpixels in the defocus distribution image is equivalent to decreasing thedensity of the defocus distribution.

In this case, Defp (x, y) is input instead of the Rp (x, y) and the g(x, y) in the Def distribution calculating unit 401 of the imageprocessing unit 206. The Def distribution calculating unit 401 generatesupdated defocus distribution image data Def′ (x, y) by substitutingpixel data corresponding to the Defe (x, y) from the image enlargingunit 402 with pixel data (defocus amount) included in the Defp (x, y).In this case, since the resolution of the Defp (x, y) is equal to theresolution of the original image, there is no need for recalculation.

This embodiment has described a case of using an image capturing unit105 having a pixel array in which a pair of photoelectric conversionunits are arranged in the horizontal direction per microlens, as shownin FIG. 2, is described. However, the present invention can be appliedalso to the case of using an image capturing unit having a pair ofphotoelectric conversion units arranged in another direction or an imagecapturing unit in which three or more photoelectric conversion units arearranged per microlens.

FIG. 9 is a view showing another example of the pixel array of the imagecapturing unit 105. Each pixel 902 has a microlens 901 and photoelectricconversion units 903, 904, 905, and 906 that are arranged (divided) inthe horizontal and the vertical direction. Here, image data that isobtained from the photoelectric conversion unit 903 group andcorresponds to the upper right focus detection pupil is expressed as RT(x, y), and image data that is obtained from the photoelectricconversion unit 904 group and corresponds to the upper left focusdetection pupil is expressed as LT (x, y) (where x and y are integers of0 or more). Further, image data that is obtained from the photoelectricconversion unit 905 group and corresponds to the lower right focusdetection pupil is expressed as RB (x, y), and image data that isobtained from the photoelectric conversion unit 906 group andcorresponds to the lower left focus detection pupil is expressed as LB(x, y) (where x and y are integers of 0 or more). The image capturingunit 105 outputs R (x, y), T (x, y), and basic image data g (x, y) inthe following formulas as captured image data sets.

L(x, y) = LT(x, y) + LB(x, y) R(x, y) = RT(x, y) + RB(x, y)T(x, y) = LT(x, y) + RT(x, y) B(x, y) = LB(x, y) + RB(x, y)g(x, y) = L(x, y) + R(x, y) = T(x, y) + B(x, y)

The L (x, y) and the R (x, y) are images equivalent to image dataobtained when the pair of photoelectric conversion units arehorizontally arranged. Further, the T (x, y) and the B (x, y) are imagesequivalent to image data to be obtained when the pair of photoelectricconversion units are vertically arranged. Further, while theaforementioned Rp (x, y) is extraction data obtained when the pair ofphotoelectric conversion units are horizontally arranged, extractiondata obtained in the same manner when the pair of photoelectricconversion units are vertically arranged is referred to as Tp (x, y).

In this case, extraction data to be recorded or transmitted to anexternal device by the digital camera 100 is the Rp (x, y) and the Tp(x, y), as shown in FIG. 10. Further, bokeh-added reduced basic imagedata Vs (x, y) is determined by combining:

-   -   Vsr (x, y) based on the reduced defocus distribution image data        Defs (x, y) determined based on the g (x, y) and Rs (x, y); and    -   Vst (x, y) based on the reduced defocus distribution image data        Defs (x, y) determined based on the g (x, y) and Ts (x, y). The        same applies also to reduced defocus distribution image data        Defs (x, y). The method for combining a plurality of image data        sets may be achieved by adding and averaging, for example.

Since two pairs of image data sets are processed, FIG. 10 shows aconfiguration in which an image processing unit 107′ having the sameconfiguration as the image processing unit 107 is added, but this is forconvenience of the description and for facilitating understandingthereof, and thus the addition of the image processing unit 107′ is notessential.

Further, an image processing unit 206′ having the same configuration asthe image processing unit 206 is similarly added also for generation ofbokeh-added basic data, as shown in FIG. 11. Then, bokeh-added basicimage data Vr (x, y) obtained from the Rp (x, y) and bokeh-added basicimage data Vt (x, y) obtained from the Tp (x, y) are combined so thatbokeh-added basic image data V (x, y) is generated. The combining, forexample, may be achieved by adding and averaging.

In the case where only focus detection information Rp (x, y) in thehorizontal direction is provided, horizontally striped objects cannot beappropriately processed. In contrast, use of both the focus detectioninformation Rp (x, y) in the horizontal direction and focus detectioninformation Tp (x, y) in the vertical direction as shown in FIG. 11enables the bokeh-added basic image data V (x, y) in which verticallystriped objects and horizontally striped objects are both appropriatelyprocessed to be output.

Second Embodiment

Next, an image processing system according to a second embodiment of thepresent invention will be described. The image processing system in thisembodiment can be constituted by the digital camera 100 and the PC 200described in the first embodiment, and therefore redundant descriptionswill be omitted.

FIG. 7 is a block diagram showing an example of a functionalconfiguration of the image processing unit 107′ included in the digitalcamera 100 in this embodiment. In FIG. 7, the same components as in FIG.4 are denoted by the same reference numerals. The image processing unit107′ of this embodiment does not have the image enlarging unit 313 andthe region extracting unit 312 in the first embodiment.

In this embodiment, the digital camera 100 generates the following dataas data to be recorded.

-   -   Basic image data g (x, y) (6400×4800 pixels)    -   Image data R (x, y) corresponding to the right focus detection        pupil (6400×4800 pixels)    -   Reduced defocus distribution image data Defs (x, y) (1600×1200        pixels)

Accordingly, in this embodiment, the amount of data to be recorded inthe recording medium 108 or transmitted from the communication unit 109to the PC 200 is greater than in the case of transmitting the R (x, y)and the L (x, y).

However, since the calculation of the defocus amount is performed basedon the reduced image data, the processing load of the Def distributioncalculating unit 304 can be significantly lightened. Further, theprocessing implemented by the image enlarging unit 313 and the regionextracting unit 312 can be omitted.

FIG. 8 is a block diagram showing an example of a functionalconfiguration of the image processing unit 206′ included in the PC 200in this embodiment. In FIG. 8, the same components as in FIG. 5 aredenoted by the same reference numerals. The image processing unit 206′of this embodiment is different from the first embodiment in a part ofoperations of the Def distribution calculating unit 401′ and the bokehadding unit 407′.

The Def distribution calculating unit 401′ in this embodiment determineswhether or not the defocus amount indicated by the defocus distributionimage data Defe (x, y) obtained from the image enlarging unit 402 isincluded in the range of the defocus amount in which the bokeh addingunit 407′ controls a bokeh amount. This determination is the same as thedetermination of the necessity of recalculation (update) in the firstembodiment. Then, in the case where it is determined that therecalculation (update) of the defocus amount is necessary, the Defdistribution calculating unit 401′ recalculates the defocus amount.

In this embodiment, the R (x, y) is input instead of the Rp (x, y), butthe basic calculation method is the same as in the first embodiment. TheDef distribution calculating unit 401′ uses image data L (x, y) thatcorresponds to the left focus detection pupil and is obtained bysubtracting image data R (x, y) that corresponds to the right focusdetection pupil from the basic image data g (x, y), instead of the Lp(x, y) in the first embodiment.

Also in this embodiment, the defocus amount is recalculated only forpixels whose bokeh amount is controlled by the bokeh adding unit 407′,and therefore the processing load for calculating the defocus amount canbe significantly lightened. Further, the recalculated defocus amountshows higher resolution than the defocus amount obtained from the imageenlarging unit 402, and therefore high-accuracy bokeh adding processingcan be realized.

In the same manner as in the first embodiment, the maximum value kmax ofthe displacement amount k is adjusted for each pixel, thereby shorteningthe time to calculate the correlation amount C(k) in pixels having smalldefocus amounts, and therefore the processing can be implemented at highspeed.

In this embodiment, the Def distribution calculating unit 401′ receivesthe entire image data R (x, y) that corresponds to the right focusdetection pupil, instead of the Rp (x, y) that is a part of the R (x,y). Then, by subtracting the R (x, y) from the basic image data g (x,y), the Def distribution calculating unit 401′ restores the entirety ofthe image data L (x, y) 701 that corresponds to the left focus detectionpupil. The L (x, y) 703 is used for recalculating the defocus amount,and is output to the bokeh adding unit 407′.

The bokeh adding unit 407′ carries out the bokeh adding processing onthe R (x, y) and the L (x, y) using the same method applied to the basicimage data g (x, y) in the first embodiment and outputs image data VR(x, y) 702 and VL (x, y) 703 resulting from the bokeh addition. An adder706 adds the VR (x, y) 702 and the VL (x, y) 703, and generates andoutputs bokeh-added basic image data V (x, y) 410.

In the first embodiment, the bokeh-added basic image data V (x, y) isgenerated by performing the bokeh adding processing after the basicimage data g (x, y) is generated. In contrast, in this embodiment, afterthe bokeh adding processing is applied separately to the image data R(x, y) that corresponds to the right focus detection pupil and the imagedata L (x, y) that corresponds to the left focus detection pupil, theyare added to each other so that the bokeh-added basic image data V (x,y) is generated.

The bokeh in the pupil-divided image data (image data that correspondsto the right focus detection pupil and the left focus detection pupil)depends on the individual optical transfer functions. Therefore, bokehaddition with higher accuracy can be achieved by generating the basicimage data after applying the bokeh adding processing to thepupil-divided image data (R (x, y) and L (x, y)) at such a stage, ascompared to applying the bokeh adding processing to the basic imagedata. This effect applies not only to the case of applying the bokehadding processing but also to the cases of applying image processingthat depends on the optical transfer function for forming apupil-divided image.

In this embodiment, the data generated for recording by the digitalcamera 100 may be another combination. For example, the image data L (x,y) that corresponds to the left focus detection pupil may be usedinstead of the image data R (x, y) that corresponds to the right focusdetection pupil. Further, a combination of the image data R (x, y) thatcorresponds to the right focus detection pupil, the image data L (x, y)that corresponds to the left focus detection pupil, and the reduceddefocus distribution image data may be used. In this case, in processingin which the g (x, y) is needed, the g (x, y) may be generated byaddition of the L (x, y) and the R (x, y).

According to this embodiment, although the amount of data for recordingand communication slightly increases as compared to that in conventionalmethods, the processing load for calculating the defocus amount can besignificantly reduced and the overall processing speed can be increased.Further, since the basic image data resulting from the bokeh processingis generated from the pupil-divided image data that has undergone thebokeh processing, the accuracy of the bokeh adding processing can beimproved as compared to the method of the first embodiment.

This embodiment also can be applied to the case of using an imagecapturing unit having a pair of photoelectric conversion units arrangedin another direction different from the horizontal direction or an imagecapturing unit in which three or more photoelectric conversion units arearranged per microlens, by being configured as shown in FIG. 10 and FIG.11.

Exemplary embodiments of the present invention have been describedabove, but the present invention is not limited to these embodiments,and various modifications and changes can be made within the rangespecified in the scope of claims.

Other Embodiments

In the aforementioned embodiments, descriptions are given on theassumption that image processing to be performed according to thedefocus amount on the basic image data (non-reduced image data) isimplemented by the image processing apparatus that is separate from theimage capture apparatus. However, the image processing may beimplemented by the image capture apparatus, for example, byincorporating the image processing apparatus in the image captureapparatus, or allowing the control unit 101 of the image captureapparatus to perform the functions of the image processing apparatus.Also in such cases, the effect of cutting the amount of data forrecording can be achieved. For example, in the case where thecommunication unit 109 cannot communicate with the image processingapparatus normally (for example, in the case of poor wirelesscommunication quality), the image processing may be implemented in thedigital camera 100. The processing time will be longer than in the caseof the implementation in an external image processing apparatus, but afinal image can be created even in a non-communicable environment.

Further, the aforementioned embodiments describe a configuration inwhich a pair of captured image data sets that are obtained by an imagesensor, whose pixels each have a plurality of photoelectric conversionunits, receiving luminous fluxes that have passed through differentregions on the exit pupil, and the pair of captured image data sets areused for calculating the defocus amount. However, the pair of capturedimage data sets that can be used for calculating the defocus amount maybe obtained using another method. For example, the pair of capturedimage data sets may be a pair of captured image data sets obtained by astereo camera or a pair of captured image data sets obtained by shootingwhile the focus distance is changed. In such a case, the aforementionedprocessing may be performed using one of the pair of sets of image dataas the R (x, y) and the other thereof as the L (x, y). That is, theconfigurations of these embodiments are applicable also to a digitalcamera including an image sensor without the pupil dividing function.

Further, in the aforementioned embodiments, a case was described inwhich bokeh adding processing is performed as the image processingapplied according to the defocus amount, and therefore the point spreadfunction is also configured to implement a low-pass filter for thepurpose of bokeh addition. However, other image processing can beperformed. For example, in image processing for sharpening an image byapplying a band-pass filter, the properties of the band-pass filter maybe changed using a parameter that corresponds to the defocus amount. Inthis way, even in the case where the amount of light is insufficient andthus the aperture opening needs to be widened in order to performshooting, image data with a large depth of field can be obtained fromthe captured image data.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiments and/or that includes one or morecircuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiments, and by a method performed by the computer of the system orapparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiments and/or controlling theone or more circuits to perform the functions of one or more of theabove-described embodiments. The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-Ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-095505, filed May 2, 2014, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image capture apparatus comprising: areduction unit configured to reduce a plurality of sets of capturedimage data including pupil-divided image data, which is captured imagedata corresponding to part of an exit pupil, and generate a plurality ofsets of reduced image data; a generation unit configured to, on at leasta pair of the plurality of sets of reduced image data that includereduced image data of the pupil-divided image data, perform processingin which a pair of sets of reduced image data are used to generatedistribution image data that indicates a spatial defocus distribution;an extraction unit configured to, based on the distribution image data,extract data of pixels that are to be subjected to predetermined imageprocessing from the pupil-divided image data corresponding to thedistribution image data; and an output unit configured to output theextracted pixel data, the distribution image data, and captured imagedata corresponding to the entirety of the exit pupil.